Using the Crowd to Monitor the Cloud: Network Event Detection from Edge Systems

The user experience for networked applications is becoming a key benchmark for customers and network providers when comparing, buying and selling alternative services. There is thus a clear need to detect, isolate and determine the root causes of network events that impact end-to-end performance and the user experience so that operators can resolve such issues in a timely manner. We argue that the most appropriate place for monitoring these service-level events is at the end systems where the services are used, and propose a new approach to enable and support this: Crowdsourcing Cloud Monitoring (C2M). This paper presents a general framework for C2M systems and demonstrates its effectiveness using a large dataset of diagnostic information gathered from BitTorrent users, together with confirmed network events from two ISPs. We demonstrate that our crowdsourcing approach allows us to detect network events worldwide, including events spanning multiple networks. We discuss how we designed, implemented and deployed an extension to BitTorrent that performs real-time network event detection using our approach. It has already been installed more than 34,000 times.